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server ( http://ipr.etsi.org) . 

Pursuant to the ETSI IPR Policy, no investigation, including IPR searches, has been carried out by ETSI. No guarantee 
can be given as to the existence of other IPRs not referenced in ETSI SR 000 314 (or the updates on the ETSI Web 
server) which are, or may be, or may become, essential to the present document. 



Foreword 

This Technical Specification (TS) has been produced by ETSI User Group (USER). 



Introduction 

The present document provides the requirements needed to ensure that QoS information is assessed according to the 
best practices as detailed in EG 202 843 [i.l]. It is closely linked to: 

• EG 202 843 [i.l] that defines parameters and basic information which are universally applicable to QoS 
assessment. One aim here is to keep the parameter definitions stable and complete for any kind of application. 

• TS 102 844 [1] that defines when, how and by whom a conformity assessment audit of the QoS assessment 
shall be conducted, if such conformity assessment is desired. 
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Scope 



The present document defines a common set of test sequences that are appHcable by any stakeholder in order to provide 
results that are objective, reproducible and comparable. 

It details - in a generic way - the assessment sequences for the 4 types of parameters: Time, Percentage, Number and 
Opinion Ratings as given in EG 202 843 [i.l]. 

The present document also provides examples of a set of dedicated assessment sequences for parameters of selected 
stages of the customer relationship course (QoS Assessment process). 



2 References 

References are either specific (identified by date of publication and/or edition number or version number) or 
non-specific. For specific references, only the cited version applies. For non-specific references, the latest version of the 
reference document (including any amendments) applies. 

Referenced documents which are not found to be publicly available in the expected location might be found at 
http://docbox.etsi.org/Reference . 

NOTE: While any hyperlinks included in this clause were valid at the time of publication ETSI cannot guarantee 
their long term validity. 

2.1 Normative references 

The following referenced documents are necessary for the application of the present document. 

[1] ETSI TS 102 844: "User Group; Quality of Telecom Services; Conformity assessment; 

Requirements for bodies providing QoS assessments and surveys". 

2.2 Informative references 

The following referenced documents are not necessary for the application of the present document but they assist the 
user with regard to a particular subject area. 

[i.l] ETSI EG 202 843: "User Group; Quality of ICT Services; Definitions and Methods for Assessing 

the QoS parameters of the Customer Relationship Stages other than utilization" . 

[i.2] ITU-T Recommendation P. 832: "Subjective performance evaluation of hands-free terminals". 

[i.3] ETSI EG 202 057: "Speech and multimedia Transmission Quality (STQ); User related QoS 

parameter definitions and measurements". 

[i.4] ETSI EG 202 009: "User Group; Quality of telecom services" . 



3 Definitions and abbreviations 

3.1 Definitions 

For the purposes of the present document, the following terms and definitions apply: 

test method: assessment method of QoS parameters carried out by a Quality of Service Assessment Party (QoSAP) 
providing QoS audits and surveys based the assessment of parameters described in EG 202 843 [i.l] 
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test sequence: detailed procedure of the test method applying to the types of parameters as described in 
EG202 843[i.l] 

NOTE: Test in the context of the present document and EG 202 843 [i.l] has a different meaning from 
compliance or interoperability testing. 

3.2 Abbreviations 

For the purposes of the present document, the following abbreviations apply: 

ADSL Asymetric Digital Subscriber Line 

CS Customer Survey 

DSL Digital Subscriber Line 

EP Expert Panel 

FTTH Fiber To The Home 

IPTV Internet Protocol Television 

ISDN Integrated Services Digital Network 

KPI Key Performance Indicator 

OR Opinion Rating 

PDA Personal Digital Assistant 

PI Preliminary Information 

PSTN PubHc Switched Telephone Network 

QoS Quality of Service 

QoSAP Quality of Service Assessment Party 

SMS Short Message Service 

SP Service Provider 

SPDA Service Provider Data Analysis 

SPO Service Provider Oriented (parameter) 

TMF Telemanagement Forum 

VoD Video on Demand 

VoIP Voice over IP 



4 Definition of a QoS assessment campaign, aims and 

purposes 

Prior to executing a campaign, the concrete aim of the campaign shall be defined and documented. The campaign may 
be initiated by any stakeholder (SP, Consumer association, regulatory bodies, etc.). The design of the campaign could 
depend on the specific objectives of a stakeholder. The next clauses define generic principles to design a campaign. The 
QoS parameters shall be implemented according to EG 202 843 [i.l]. 

4.1 General 

The EG 202 843 [i.l] defines the different stages of the customer relationship, the parameters and the preferred methods 
to assess them. Table A. 1 summarizes the main information and recommends the preferred approaches detailed in 
EG 202 843 [i.l]: Customer Survey, Expert Panel and SP Data Analysis. 

Defining the aim of a campaign means that several choices and decisions have to be carried out. One of the important 
initial tasks is to define the perimeter of the campaign and to define the profile of the panels to participate to the 
campaign. 

4.2 Questions to be answered when designing a QoS 
assessment campaign 

When designing a QoS assessment campaign, a set of questions has to be answered, which determine the elements to be 
taken into account within the assessment campaign. 
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Several Service Providers may be involved in the assessment campaign. For each of them the following questions shall 
be answered: 

What are the selected stages of the customer relationship lifecycle, as defined in EG 202 843 [i.l]? 

What is the list of services within the SP offers that will be assessed? 

Phases can be used as modules; different modules building the complete interrogation. 

How many Service Providers will be involved in the assessment campaign? 

Is the assessment campaign defined as a Benchmarking? -Direct comparison of different providers 
(e.g. comparison of success rates or comparison of time parameters across different providers). 

Will the assessment campaign be a "one shot" or be made periodically? 

What type of approach has to be implemented: customer survey, expert panel, SP Data, Analysis or a mix of 
them? 

What customer group has to be involved (For customer survey and expert panel, the profiles and the number of 
participants have to be specified). 

4.3 Tasks for the campaign 

A table defining a set of tasks to be done by the Expert Panel is provided in informative annex E. 

• For the expert panel the tasks are designed specifically in order to ensure all the providers and/or services are 
treated in a similar way and to reduce as much as possible the time spent in collaborative works between the 
experts. 

The customers involved in the customer survey shall be currently using the services and they have to answer a set of 
questions. 

• For customer survey real example of a customer survey campaign including several customer relationship 
stages is provided in annex D and could be used as an informative reference when defining a campaign. 

• For the approach based on SP Data Analysis the design of the test should take into account the type of data 
made available by each service provider. 



Boundary conditions for campaign 



For recurrent and benchmarking campaigns the conditions have to be clearly defined in order to make results 
comparable and reproducible. This includes in particular the selection of expert panels or customers participating in the 
surveys, the test duration, the media to access participants, etc. 

When preparing an assessment campaign related to the customer relationship course all the variables that may influence 
the assessment outcomes shall be defined. One main reason for detailing all the parameters is to give the possibility for 
other parties to reproduce comparable assessments. As soon as one of the variables is changed, the campaign might 
characterise a different assessment with deviations in the results. 

All the parameters that define an assessment campaign have to be checked carefully and listed in the assessment plan, in 
particular: 

• The parameters that are retained within each stage have to be defined for each stage of the customer 
relationship life cycle. This could include dedicated part of customer relationship cycle, e.g. only DSL clients, 
only roaming issues of business users, etc. 

• As EG 202 843 [i.l] defines three types of approaches, "Expert panel", "Customer panel" or "Service Provider 
data Analysis", the most appropriate type must be chosen for the different parameters (see annex A). For each 
parameter included in the campaign, the option chosen has to be defined and justified. 
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If the campaign includes Customer surveys, the appropriate customer groups to be involved in the assessment 
shall be determined. In particular it shall be defined if there is a focus on private and/or business customers. 
The number of customers involved in the campaign shall also be established to ensure statistically valid 
results. It shall be checked that the customers involved in the assessment sessions will not participate in more 
than 2 sessions per year. 

If the campaign includes expert panels, the characteristics/profiles of the experts to be involved shall be 
defined. 

The assessment plan shall also define the media that will be used to address the attendees and the media that 
are foreseen for the feedback of the attendees. 

The assessment plan shall define the campaign duration, and if the campaign is carried out on a periodical 
basis and the periodicity of the sessions. 

If questionnaires are addressed to customers, it shall be defined if and when reminder messages could be sent 
during the runtime of the assessment. 

It shall be checked if there is a special geographical area of interest, e.g. handling of DSL contracts in rural 
areas where no broadband data service is available due to physical constraints. 

For customer surveys, it shall be defined what kind of methods will be used: e.g. telephone interview, mailing 
activity, online questionnaire, direct interview in a shopping mall. 

For expert panel it shall be defined if the experts will be questioned in-situ or at distance, and if they will do it 
individually or all together (in such a case, preferably a physical meeting). 

The operational team (QoSAP) that will design and run the assessment shall be defined and documented. This 
shall be defined for customer surveys, expert panels and analysis of SP Data. 

It shall be checked if the operational team (QoSAP) needs to involve other companies or departments, e.g. call 
centres for phone interrogations, companies for recruiting panels, companies for campaign execution. 

For Expert panel and Customer survey even if the approaches may be different, the questionnaires/measurement 
methods may be rather similar. 

In particular, questionnaires shall be designed to select customers and experts to be involved in the assessment sessions. 
These questionnaires should be based on the contents of annex C. 

It shall be taken into account that the customers may only reply for services or providers they have actually 
experienced. 

For experts (or experienced/trained users) it is needed to design the tasks so that all the service providers are assessed in 
similar/equal conditions and equal footing. The experts are participating in the assessment session for a limited period 
of time and they shall complete the tasks which are defined within the assessment sequences. 



5.1 Customer survey 



To get a broader basis of feedback, customer surveys (panels) can be used. A customer panel consists of common 
customers of products or services. The customers should be familiar with the topic they are surveyed for, but without 
reaching an expert level. The selection should be based on replies to the relevant questions defined in annex C. The 
selection may also be done by Service providers, customers groups, etc. 

In many cases, specialized institutes are engaged to deal with the panel recruitment. This is based on the fact that either 
a well-defined part of the population should be taken into consideration (e.g. only females aged 25 to 35 years with a 
certain net household income) or that the selected group of customers should be representative for the complete 
population of this country or for the complete population of customers of a service provider. 

When selecting the customer panel it may be needed to ask questions related to the user's background, such as defined 
in annex C. 

To ensure relevant statistical results, the number of customers shall be important. The level of customer attendance to 
ensure the desired level of statistical relevance of data is defined in EG 202 843 [i.l] and copied in clause 5.4 of the 
present document. 
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When designing the assessment event it is needed to take into account that additional expenses are generated by the 
involvement of market research institutions. 

Customer survey interrogations need currently a longer period of time than for the expert panel (up to several weeks). 

When Opinion Rating (OR) is sought via both a customer survey and an expert panel, there may be discrepancy 
between the findings of these differing means. Where the difference is significant, reason for this discrepancy should be 
investigated and any necessary changes incorporated either to the panel's ratings or the way the customer survey is 
carried out. 

When questionnaires are sent to an email list of customers (or by postal mail) or when customers are asked by phone 
calls, the percentage of replies may be very low compared to the whole customer list. This has to be taken into account 
to ensure a high level of statistical relevance of the results. 



5.2 Expert panel 



An expert panel is defined as a group of experts, who are very familiar (or professional) with the topic of interest. The 
expert panel evaluates the topic of interest and gives expert opinion on it. 

For some experiments, experts may be replaced by experienced or trained customers/users. To distinguish experts and 
experienced/trained users, the following definitions have been adapted from ITU-T Recommendation P. 832 [i.2]: 

• Experienced/trained users: Experienced/trained users (for the purpose of customer relationship QoS 
evaluation) are experienced in QoS testing and analysing the quality of some stages of the customer 
relationship, are not individuals who routinely conduct such assessments. Experienced/trained users are able to 
describe their subjective impressions on the different stages in detail. However, experienced/trained users 
neither have a background in technical implementations of customer relationship stages covered by 

EG 202 843 [i.l] nor do they have detailed knowledge of the influence of the parameters on the quality 
perceived by the customers. 

• Experts: Experts (for the purpose of customer relationship QoS evaluation) are experienced in QoS testing 
and analysing the quality of the different stages of the customer relationship. Experts are able to describe all 
the events, tasks and deliverables of the customer relationship in detail and are able to implement the 
assessment methods or to analyse the results as described in EG 202 843 [i.l] and to analyse the specific 
impairments in the processes. They are able to describe their impressions in detail. They have a background in 
technical implementations of the customer relationship stages and/or do have detailed knowledge of the 
influence of customer relationship implementations on QoS perceived by the customers. Experts may be used 
for all types of assessments defined in EG 202 843 [i.l]. Typically the expert's judgement is validated by 
untrained users representing the average user group the set is intended to be used for. 

In case where the number of experts cannot be reached, experienced/trained users may complete the team or the group. 
However, if experts and experienced/trained users are mixed within an experiment care should be taken to avoid experts 
masking the feedbacks from experienced/trained users, in particular when all the participants meet together. 

Ideally, the selected experts bring a broad theoretical background and practical experience as well as a longer period of 
personal knowledge with them. Besides that the selection of experts should take into consideration that all relevant 
aspects of the examined topic are covered by the combination of experts within the panel. 

The number of experts required to address a certain topic should be limited (currently between 10 and 20). 

A questionnaire to select expert panel is defined in annex C: 



• 



Selection process and criteria for selection are very important because the quality of the experts may influence 
severely the quality of the results. 

• It is also important that all required experts may be together at the same place and time (or over a short period 
of time), or by high quality conference systems. 

NOTE: It is needed to plan for expenses generated by the involvement of experts. 
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For the expert panel, two different approaches can be undertaken, depending of the relationship stage: 



• 



• 



In a first step each expert realises specific tasks individually and provides replies to questionnaires and 
comments based on their own knowledge and perceptions; in a second step they meet together to exchange on 
the results or observations obtained. 

The task is done collectively on the basis of materials made available to all the experts during a specific 
meeting. In such a case feedback to one specific subject can be collected rather quickly (during an experts' 
meeting). 

As Experts are used as highly-trained customers they may reflect customers' point of view and may also comment 
together on the results. 

Studies carried out on particular QoS aspects like assessment of call centre QoS made using "mystery calls" or QoS of 
mobile communications by human operators belongs to this category of data source. The experts involved in the 
"mystery calls" should act as normal customers but shall ask pertinent questions to obtain detailed answers. 

5.3 Service Provider Data Analysis 

For certain customer relevant processes, service providers (SPs) may have available customer records for their own 
purposes or due to regulatory requirements. In these cases such data may be used for the determination of customer 
relevant parameters as well but in a well controlled process. 

NOTE: It appears that some parameters defined by Telemanagement Forum (TMF) may be used for such 
approaches. Some further investigations will be needed to achieve this. 

Before using SP Data for an analysis, it shall be checked that the parameters and measurement methods defined in 
EG 202 843 [i.l] apply for the data collected. If the measurement conditions implemented by the Service Providers 
differ from those available in EG 202 843 [i.l], it should be checked that the data may be used without any trouble. The 
service provider shall document the measurement conditions for the data collection and shall ensure that the purpose 
and conditions are compliant with the principles defined EG 202 843 [i.l]. Prior to implementing such a process it will 
be checked that the SP Data are well correlated with parameters defined in EG 202 843 [i.l]. 

It could be expected that such an approach should reduce the data generation since the data is available from the usual 
day-to-day business. It could also be associated to customer surveys or expert panels. 

A large amount of data sets may be available (mass data), depending on the number of customers contracted by the SP 
and depending on their activity. 

For such an approach the service providers will be an active actor of the process. 

5.4 Samples sizes and examples (excerpt from EG 202 843) 

Besides the different nature of the mentioned data sources, the number of available samples for each of these data 
sources may also differ heavily, as indicated in EG 202 843 [i.l]. 

• To assess a special topic, only a few but highly trained experts are required. This leads to a high quality 
feedback, but includes also very limited number of information. 

EXAMPLE 1: 15 experts are requested to assess the "Integrity of Complaint Resolution". The outcome will be 15 
different opinions on corresponding scale. 

• The assessment of topics which are more common to all customers and which do not require special expertise 
allows the involvement of a higher number of customers. 

EXAMPLE 2: Replies from 150 customers of SP A who complained about a certain matter are selected to give 
their feedback on the "Customer Perception of the Complaint Management". 

Here, the quality of the feedback will not be on expert level, but represents the customer perception very 
clearly. Furthermore, the number of samples is higher than in the first case which improves the data basis for 
statistical operations. 
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• Finally, if mass data from service provider's internal processes can be assessed, there are two advantages: The 
weight of each data set on the overall result is negligible, and most of the data will be measured objectively. 

EXAMPLE 3: SP B delivers 10 000 data sets which allow to determine the parameter "Time for alteration" on a 
very broad basis. 



Preparation of the campaign 



After defining the basis of the assessment by fixing the basic boundary conditions, the operational phase should be 
prepared, depending on the assessment approaches chosen. 

The following should be born in mind when planning surveys. 

Basically there are three methods of carrying out a customer survey: 

1) questionnaire posted to the interviewees; 

2) questionnaire administered by an interviewer over telephone; and 

3) administering questionnaire on a face to face basis between the interviewer and the interviewee. 
Each method has its merits and disadvantages. These are outlined briefly here. 

Questionnaire posted to the interviewee 

Respondents are asked to fill in the quality ratings for the performance parameters. 

Advantages: 

Questionnaire is a powerful tool to capture customers' performance ratings. This method of administering a survey is 
also the cheapest as very little manpower is involved. 

Disadvantages: 

In filling in questionnaires, the respondent does not usually have the means to seek clarification on questions they do 
not understand. This could result in vague answers. Some parts of the questionnaires may be left blank. There may also 
be some questions the respondents may have misunderstood. 

The response rate is low. In general these vary from below 5 % to upper 30's % for user club members. This means a 
very large sample size has to be sent the questionnaire to obtain a statistically valid number of responses. Additionally 
the answers should be carefully analysed to ensure that the replies provided clearly indicate that the customer has 
understood the question. It must be borne in mind that some customers fill in the form without much thought. Indeed in 
certain psychological questionnaires it is common to insert a "lie detector" question to ensure that the answers provided 
are consistent to the person's beliefs in providing answers. 

Questionnaire administered by an interviewer over telephone 

Advantages: 

Telephone questionnaires are a compromise between postal questionnaire and face to face interview. The interviewer 
can ensure to a greater extent than the postal survey that the customer has understood the question. There is scope for 
the customer to clarify points she/he is unsure of. The response rate is also much higher than those of postal survey. 

Disadvantages: 

The principal shortcoming of this type of interview is the absence of body language of the respondent. This limits the 
understanding of the questioner of the respondent's concerns. 

Cost of administering telephone interviews are higher due to the manpower costs comprising overhead and training 
costs. 
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Administering questionnaire on a face to face basis between the interviewer and the interviewee 

Advantages: 

Face to face interviews, with a questionnaire, should provide the ideal alternative to the postal and telephone 
questionnaire. Some of the shortcomings mentioned in the previous methods can be eliminated. 

Face to face interviews provide the most credible answers and therefore the confidence in their accuracy and relevance 
is very high. This type of survey is usually kept aside for large organisations. 

Disadvantages: 

Face to face interviews are time consuming and make a high demand on resources.* 

NOTE: The sample size required for postal questionnaire is the highest, with telephone interviews requiring fewer 
samples and the method of personal interviews requiring least samples for a given level of confidence. 



6.1 Customer Survey 



When the area of the campaign has been defined and the questionnaire built up it is important to prepare all the 
materials that will be addressed to the potential participants (by post, e mail, phone call, etc.). 

As indicated in annex B, the questionnaire will be addressed to the selected participants. Care should be taken when 
defining the messages associated or preceding the questionnaire. These messages shall provide confidence and show the 
professionalism of the process. 

It should also take care of security matters, in particular when using online questionnaires. 

A pre-test shall be organised to ensure or improve the overall quality of the activity: 

• Prior to official assessment campaign the pre-test phase shall be organised in the exact same way as the real 
test but with reduced number of participants. All the comments and issues received during this pretest phase 
shall be considered to finalise the assessment campaign. 

To save cost and time it is preferable to prepare automated evaluation routines that can be applied during the evaluation 
phase and after the evaluation phase has finished. 

During the experiment it is strongly recommended to monitor the development of the results; this may prevent issues 
from only being discovered after finishing the assessment campaign, e.g. if certain data is entered by the customers but 
does not reach the database due to minor errors in scripting. 

The QoSAP shall inform the participants on the availability of an hotline that may be reached by phone or mail and to 
indicate the delay expected for replies to messages addressed to the hot line. 

Care shall be taken to ensure the availability of all required resources, e.g. agents in the call center, mailing and 
computer systems for sending out invitation emails and for collecting feedback information. 

The QoSAP shall prepare a monitoring of all relevant activities within the evaluation process, at least: 

• To set up a quick reaction team to allow last minute changes within the assessment if this scenario might be 
useful. 

• To monitor all kind of user activity on the system (mainly for online questionnaires): 

To prevent running into overload situations (processor power, storage capacity of database, data links, 
etc.). 

To prevent participants from misusing the system. In particular the injection of query statements to 
databases and the addition of remotely controlled frames to web applications should be checked in 
advance. 

The QoSAP shall define a backup strategy to prevent from data loss. 
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6.2 Expert panel 



For Expert panel and Customer survey the approaches are different. The expert panel works on a task-oriented process, 
as defined in annex E. However, some questions addressed to experts may be similar or based on examples available in 
annex D. 

Due to the reduced number of experts compared to customers involved in a survey, the process may be simplified. 
However processes similar to customer surveys may be used for communications with the experts and to collect the 
data. 

When experts are invited to meet physically or through high quality teleconference systems, the scenario of the meeting 
shall be defined very precisely, in particular the tasks to be done collectively and it shall be needed to ensure that all the 
experts are placed in similar conditions and on equal footing (all the experts have to express themselves; it shall be 
avoided that one or some experts take the lead of the discussions and impose their views). For additional analysis, the 
meeting may be recorded (audio and/or video). 



6.3 Analysis of SP data 



After having checked that the SP data may be used to assess some parameters defined in EG 202 843 [i.l], the service 
provider shall provide data in formats that are defined by the QoSAP. 



7 Execution of the cannpaign 

The execution of the campaign includes: 

• The execution of the monitoring (see annex B). 

• The activation of the hotline team for competent and quick resolution of issues. 

7.1 Customer survey 

It is not intended to invite customers to any meeting to discuss the comments or the additional information. 

After the questionnaire has been sent to the participants, they have to reply before the defined deadline (usually two 
weeks). It may be relevant to send a reminder after one week to the potential participants who did not already answer. 

When sending the questionnaires by email, several preventive actions have to be taken to avoid unsecure exchanges. In 
particular when using online questionnaire, the access to this questionnaire should be given in a two step process 
including email exchanges. An example is given in annex D. 

It is also important during the campaign execution to ensure a daily hotline to reply to questions or remarks from the 
participants. The media used for the hotline should be as close as possible to the media used for the questionnaire. 



7.2 Expert panel 



The experts will be invited to a meeting to discuss the results they individually obtained or to work all together in a 
collaborative way. 

If experts have to work individually before the meeting, they shall have to complete these actions prior to the meeting. 
The duration of these preliminary actions will be defined according to the tasks themselves. Prior to these actions the 
QoSAP shall have prepared all the materials needed by the experts. During these periods the QoSAP shall be able to 
answer quickly to questions or messages from the experts and to do all the relevant actions to solve potential issues. 

The date and duration of the meeting will be known sufficiently in advance and all the experts shall participate. 

The chair of the meeting shall be experienced in this kind of meeting. 
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7.3 Analysis of SP data 

The tables provided by the QoS AP have to be filled by the service provider. 



8 Data filtering 



After the operation phase has been concluded, all data should be validated before doing computational assessments. 
This comprises different tasks like: 

• Removing all unrealistic data, e.g. fake users who want to disturb your assessment. 

• Removing data which was entered due to system problems, e.g. values out of range. 

Before doing any manipulations on the data basis, all available data should be stored in a safe backup. If one of the 
following steps destroys data or manipulates data in an unintended manner, there is still a chance to recover from these 
situations. 

If data has to be removed from the data basis, this should be done by setting an invalid marker to the data instead of 
physically deleting the data. The ratio of invalidated and validated data sets can be interpreted as an additional quality 
measure for the evaluation. 



Aggregation of results 



According to the pre-specified rules, data now can be aggregated. The aim of this activity is to transform the retrieved 
data to key performance indicators (KPIs) which give a highly condensed view of the underlying data. 

Typical aggregation dimensions are time, location, addressed customer group, related provider, etc. 

Besides the mean value of some statistically distributed data further methods should be taken into consideration: 

• Calculation of minimum and maximum values. 

• Quantile values, typically 5 % and 95 % quantiles which give some outlier related information. 

• Compressed footprints of collected data by giving a set of quantile values, e.g. 5 %, 10 %, 50 % (median), 
90 % and 95 % quantiles, extended by the mean value. 

Great differences between mean and median value should be examined in detail. This scenario often represents large 
outlier values with a non-negligible influence on the mean value (whereas the median value is not influenced by these 
outliers due to its robustness). 



1 Analysis of the results of the assessment campaign 

To combine stand-alone KPIs to an integrated judgement, it should be appropriate to have a closer look at it to focus on 
root causes and their effects. This phase is necessary to establish a link to further optimisation activities to improve the 
overall situation for the customer. 

This phase has to be conducted by QoS AP with an in-depth knowledge. They add their interpretation of the observed 
data to the KPIs. These additional comments allow an easier understanding of single issues as well as a more complete 
overall picture. In the end, KPIs with comments will give a readable report for non-expert readers. 

However the statistics could be treated taking into account the differences in the numbers of participants (few for 
experts, a lot of for customers). 

When several assessment methods (SP Data Analysis, EP, CS) are implemented, the different results should be 
crosschecked. If potential coherences/incoherencies between results obtained by different assessment methods for some 
parameters are identified, these should be analyzed and reported. 
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When the assessment campaigns are conducted on a periodical basis, the evolution of the KPIs shall be checked and 
documented. 



1 1 Report 

The assessment steps and all the factors/conditions of the assessment campaign shall be reported in detail. 
In particular the report shall provide: 

• the list of customer relationship stages assessed by the campaign; 

• the list of parameters assessed and for each of them: 

the assessment method(s) implemented; 

the number and the profiles (see annex C) of customers or experts involved in the assessment campaign. 
It shall include the process to define the panel or the participants for the customer survey; 

the list of the questions defined in the questionnaires; 

the list of tasks for the participants and in particular for the experts; 

the duration of the campaign. 

if meetings are organised, the report shall detail the different steps and other organisational information; 

for customer surveys, the percentage of attendees compared with the initially defined customers. It shall be 
reported if the answering customers are representative of the expected spread of participants; 

the list of actions done by QoSAP to contact the participants, to conduct the experiment, to collect the results, 
to store and secure the data and communications; 

for SP data analysis the tables of data in formats as defined by the QoSAP; 

the process implemented for statistical analysis of the results, including explanations about withdrawn (or not 
used) data; 

the results displayed (and when needed aggregated) according to EG 202 843 [i.l]. 

The KPIs, as defined in clause 10, are characteristic for the dedicated assessment and represent the outcome of the 
overall activity. They should be reported in an easy understandable format and should point out the key findings for 
each relevant constellation, e.g. for each question and also for each provider. 

Lots of different schemes can be applied to visualise the determined KPI sets. Since in many cases the readers of such 
reports are used to have a certain representation, changes in the reporting format should be applied carefully. 

Finally, the report including some analysis and visual representations should be distributed to the stakeholders. 
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Annex A (normative): 
Parameter overview 



The table A.l provides a parameter overview of the customer relationship stages. It includes the list of stages, the list of 
parameters, the parameter number, the type of parameter and the assessment principle (by a customer survey, an expert 
panel or an analysis of SP data). 

This table summarizes the parameters defined in EG 202 843 [i.l] where the details on the assessment methods and 
parameter computation are available. 
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Stage 


Parameter name 


Parameter 
type 


3? 

ii 


> 

8 § 

(/) 3 

(/> (D 

3 


5? 

■a 

3 

2. 


Customer 
survey 


Q. 

s 


Preliminary information (PI) 


Integrity of preliminary information (PI) 


OR 


101 


EP 


Y 






Pricing transparency 


OR 


102 


CS-EP 


Y 


Y 




Availability of preliminary information (PI) 


% 


103 


CS-SPDA 




Y 


Y 


Response time for the provision of preliminary information (PI) 


t 


104 


EP 


Y 






Response time of the commercial desk 


t&% 


105 










Overall rating of the responsiveness of the service desk 


OR 


106 










User friendliness of the Internet user interface 


OR 


107 










User friendliness of the service desk operators 


OR 


108 










Contract establishment 


Integrity of the contract information 


OR 


201 


EP 


Y 






Compliance of contractual terms with preliminary information (PI) 


% 


202 


EP 


Y 






Flexibility for customization before contract 


OR 


203 


CS-EP 


Y 


Y 




Ease and flexibility to amend terms after formal contract 


OR 


204 


cs 




Y 




Response time of the commercial desk 


t&% 


205 










Delay to settle a contract 


t&% 


206 










Delay for a contract acknowledgement 


t&% 


207 










Overall rating of the responsiveness of the sales desk 


OR 


208 










Ease of the subscription process 


OR 


209 










Vendors empathy and responsiveness 


OR 


210 










Service provisioning 


Meeting promised provisioning date 


% 


301 


CS-SPDA 




Y 


Y 


Time for provisioning 


t 


302 


CS-SPDA 




Y 


Y 


Successful provisioning within specified period 


% 


303 


CS-SPDA 




Y 


Y 


Contract cancelled due to non fulfilment 


% 


304 


EP 


Y 






Completeness of fulfilment of contractual specification in the provision of a service 


% 


305 


CS-EP 


Y 


Y 




Punctuality of service provisioning 


t 


306 


CS-EP 


Y 


Y 




Punctuality of equipment delivery for service provisioning 


t 


307 


CS-SPDA 




Y 


Y 


Provisioning not complete and correct first time 


% 


308 


CS-EP-SPDA 


Y 


Y 


Y 


Provisioning time 


t&% 


309 










Overall quality of the provisioning process including the reception desk 


OR 


310 










Provider ability to match the customer's wishes for conditions of achievement 


OR 


311 










User friendliness of the means available to the customer for the operations he has to 
perform 


OR 


312 










Portage delay (when applicable) 


t&% 


313 










Proportion of problems with number portability procedures 


% 


314 
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Stage 


Parameter name 


Parameter 
type 


3? 

ii 


it 

(/) 3 

(/) (D 

3 


m 

X 

3. 

3 
(D. 


Customer 
survey 


C/) 

■0 

Q. 
fi) 

sr 


Service alteration 


Time for alteration 


t 


401 


CS-SPDA 




Y 


Y 


Successful service alteration within specified period 


% 


402 


CS-SPDA 




Y 


Y 


Completeness of fulfilment of contractual specification in the alteration of a service 


% 


403 


CS-SPDA 




Y 


Y 


Punctuality of appointments for service alteration 


t 


404 


CS-SPDA 




Y 


Y 


Punctuality of equipment delivery for service alteration [Time] 


t 


405 


CS-SPDA 




Y 


Y 


Service alteration not complete and correct first time 


% 


406 


CS-SPDA 




Y 


Y 


Conformity and success of service alteration 


% 


407 


based on 
P402 & 403 




Y 


Y 


Technical reliability of service within an agreed period after alteration 


% 


408 


CS-EP-SPDA 


Y 


Y 


Y 


Response time of the alteration service 


t&% 


409 


CS-EP-SPDA 


Y 


Y 


Y 


Overall quality of the alteration process 


OR 


410 










User friendliness of the means available to the customer for the operations he has to 
perform 


OR 


411 










Organisational efficiency of service provider to carry out service alteration (SPO) 


OR 


412 


CS-EP-SPDA 


Y 


Y 


Y 


Teclinical upgrade 


Time for technical upgrade of a service 


t 


501 


CS-SPDA 




Y 


Y 


Successful technical upgrade within specified period 


% 


502 


CS-SPDA 




Y 


Y 


Completeness of fulfilment of specification in the technical upgrade of a service 


% 


503 


CS-SPDA 




Y 


Y 


Punctuality of appointments for technical upgrade 


t 


504 


CS-SPDA 




Y 


Y 


Outage time due to technical upgrade 


t 


505 


CS-SPDA 




Y 


Y 


Technical upgrade not complete and correct first time 


% 


506 


CS-SPDA 




Y 


Y 


Conformity and success of technical upgrade 


% 


507 


Based on 
P502 & P503 




Y 


Y 


Technical reliability of service within an agreed period after technical upgrade 


% 


508 


CS-SPDA 




Y 


Y 


Overall quality of the technical upgrade process 


OR 


509 










Provider ability to match the customer's wishes for conditions of achievement 


OR 


510 










User friendliness of the means available to the customer for the operations he has to 
perform 


OR 


511 










Organisational efficiency of service provider (SP) to carry out technical upgrade (SPO) 


OR 


512 


CS-EP-SPDA 


Y 


Y 


Y 


Competence and preparedness of Service provider (SP) for technical upgrade (SPO) 


OR 


513 


EP-SPDA 


Y 




Y 


Service Support ■ Documentation 


Documentation delivery time 


t 


611 


CS-SPDA 




Y 


Y 


Availability of documentation within specified period of time 


% 


612 


CS-SPDA 




Y 


Y 


Integrity (correctness and completeness) of documentation 


OR 


613 


CS-EP 


Y 


Y 




Modes of documentation 


N 


614 


EP-SPDA 


Y 




Y 


Legibility of documentation 


OR 


615 


EP 


Y 






Overall reliability of documentation services 


OR 


616 


EP-CS 


Y 


Y 
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Stage 


Parameter name 


Parameter 
type 


3? 

ii 


it 

(A 3 

W <D 

3 


m 

X 

3. 

3 
(D. 


Customer 
survey 


C/) 

■0 

Q. 
fi) 

sr 


Service Support - Technical support 


Accessibility of the technical support 


% 


621 


CS-EP-SPDA 


Y 


Y 


Y 


Technical solutions achieved within a specified period 


% 


622 


CS-EP-SPDA 


Y 


Y 


Y 


Number of attempts before successful solution 


N 


623 


CS-SPDA 




Y 


Y 


Integrity of technical solutions 


OR 


624 


CS-EP 


Y 


Y 




Reliability of technical solutions achieved 


% 


625 


CS -SPDA 




Y 


Y 


Modes of technical support 


N 


626 


SPDA-EP 


Y 




Y 


Recognition of the customer technical request 


% 


627 










Response time of the technical support 


t &, % 


628 










Request to technical support resolution time 


t&% 


629 










Frequency of customer requests to technical support 


N/t 


630 










User friendliness of the technical support 


OR 


631 










Service Support ■ Commercial support 


Accessibility of the commercial support 


% 


641 


CS-EP-SPDA 


Y 


Y 


Y 


Commercial solutions delivery time 


t 


642 


CS-SPDA 




Y 


Y 


Commercial solutions achieved within a specified period 


% 


643 


CS-SPDA 




Y 


Y 


Integrity of solution achieved by the SP 


OR 


644 


CS-SPDA 




Y 


Y 


Modes of commercial support 


N 


645 


SPDA 






Y 


Recognition of the customer commercial request 


% 


646 










Response time of the commercial support 


t, % 


647 










Request to commercial support resolution time 


t, % 


648 










Frequency of customer requests to commercial support 


N/t 


649 










Quality of the commercial support 


OR 


650 










User friendliness of the commercial support 


OR 


651 










Organisational efficiency of commercial supp^ (SPO) 


OR 


852 


cs-spu;s^ 




Y 


Y 


Service Support - Complaint 
management 


Accessibility of the complaint management desk 


% 


661 


CS-EP-SPDA 


Y 


Y 


Y 


Recognition of the customer complaints 


% 


662 


CS-SPDA 




Y 


Y 


Complaint solutions not complete and correct first time 


% 


663 


CS-SPDA 




Y 


Y 


Integrity of complaint resolution 


% 


664 


CS-SPDA 




Y 


Y 


Customer perception of the complaint management 


OR 


665 


CS-EP 


Y 


Y 




Overall quality of the complaint management process 


OR 


666 


CS-EP 


Y 


Y 




Response time of the complaint management desk 


t&% 


667 










Customer complaints resolution time 


t&% 


668 










Frequency of customer complaints of any kind 


N/t 


669 










Professionalism of the complaint management desk 


OR 


670 










Organisational efficiency of complaint management system (SPO) 


OR 


671 


CS-EP-SPDA 


Y 


Y 


Y 
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Stage 


Parameter name 


Parameter 
type 


3? 

ii 


it 

(A 3 

W <D 

3 


m 

X 

3. 

3 
(D. 


Customer 
survey 


C/) 

■0 

Q. 
fi) 

sr 


Repair services 


Accessibility of repair services 


% 


701 


SPDA 






Y 


Successful repairs carried out within a specified period 


% 


702 


CS-SPDA 




Y 


Y 


Repairs not complete and correct first time 


% 


703 


CS-SPDA 




Y 


Y 


Punctuality of appointments for repairs(a) 


OR 


704a 


CS-SPDA 




Y 


Y 


Punctuality of appointments for repairs(b) 


t 


704b 


CS-SPDA 




Y 


Y 


Efficiency of the repair service 


OR 


705 


CS 




Y 




Fault repair time 


t&% 


706 










Frequency of customer complaints related to repair services 


N/t 


707 










Professionalism of the repair staff 


OR 


708 










Provider ability to match the customer's wishes for conditions of achievement 


OR 


709 










User friendliness of the repair service 


OR 


710 










Organisational efficiency of repair service (SPO) 


OR 


711 


CS 




Y 




IVIetering, Charging, Billing 


Accessibility of the tariff information 


% 


801 


CS-SPDA 




Y 


Y 


Successful notification of exceeding billing budget 


% 


802 


CS-SPDA 




Y 


Y 


Notification time (delay) of exceeding billing budget 


t 


803 


CS-SPDA 




Y 


Y 


Accessibility of the account management 


% 


804 


CS-SPDA 




Y 


Y 


Time to update charging information 


t 


805 


CS-SPDA 




Y 


Y 


Timeless of bill delivery 


% 


806 


CS-SPDA 




Y 


Y 


Bill delivery delay 


t 


807 


CS-SPDA 




Y 


Y 


Late notification of amount due 


% 


808 


CS-SPDA 




Y 


Y 


Modes of billing information transfer 


N 


809 


SPDA 






Y 


Bill correctness complaints 


% 


810 










Prepaid account credit correctness complaints 


% 


811 










Provider ability to match the customer's wishes for charging/billing conditions 


OR 


812 










User friendliness of the desk in charge of billing issues 


OR 


813 










Bill presentation quality 


OR 


814 










Organisational efficiency of the billing service (SPO) 


OR 


815 


CS-SPDA 




Y 


Y 
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Stage 


Parameter name 


Parameter 
type 




it 

(/) 3 

(/) (D 

3 


m 

X 

3. 
1. 

3 
(D. 


Customer 
survey 


C/) 

■0 

Q. 
fi) 

sr 


Network / Service Management by the 
customer 


Outage duration 


t 


901 


SPDA 






Y 


Frequency of outages 


N/t 


902 


SPDA-EP 


Y 




Y 


Response time for reply to requests 


t 


903 


SPDA 






Y 


Successful request response 


% 


904 


CS-SPDA 




Y 


Y 


Overall reliability of network / service management service 


OR 


905 


CS-EP 


Y 


Y 




Accessibility ot the network/service management facility ^t^5^"=!«!:^>.. „.........,, 


t, % 


906 










Response time of the operator of the network/service management facility 


t, % 


907 










Network/Service (N/S) Management access time 


t 


908 










Frequency of customer complaints related to network / service management by the 
customer 


N 


909 










Overall quality of the network/service management process 


OR 


910 










Provider ability to match the customer's wishes for network/service management 
conditions 


OR 


911 










User friendliness of the means available to the customer for the operations he has to 
perform 


OR 


912 










Organisational efficiency of netwnrk ( cjfjpYig^jjananpmpnt q^r^^inp 




^J^ 


. CS-EP 


Y 


Y 




Cessation 


Cessation acknowledgement time 


t 


1001 


CS-SPDA 




Y 


Y 


Cessation request acknowledgement 


% 


1002 


CS-SPDA 




Y 


Y 


Accessibility of the cessation facility 


% 


1003 


CS-SPDA 




Y 


Y 


Contractual cessation achieved 


% 


1004 


CS-SPDA 




Y 


Y 


Correctness and completeness in taking the customer request into account 


N&% 


1005 










Response time of the cessation facility 


t,% 


1006 










Overall quality of the cessation process 


OR 


1007 










Frequency of customer complaints related to cessation 


N 


1008 










Ease of the cessation process 


OR 


1009 










Explanation of the different colours used to identify the various parameter categories 


CS 


Consumer survey 


EP 


Expert Panel 




Parameters defined by STF 374 (customer view) 


Trained C 


Trained Consumer Panel 




SPDA 


SP Data Analysis 




Predefined parameters (e.g. EG 202 057 [i.3] or EG 202 009 [i.4]) 


In bold the preferred scenario where 




Parameters defined by STF 374 (service provider view) 


appropriate 
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Annex B (normative): 

Questionnaires for customer survey and expert panel 

This annex applies only for customer survey and expert panel. It does not apply for analysis of SP Data. 
In general, most of the parameters to be assessed can be covered by four types of parameters: 

Opinion Rating (OR). 

Rate (%). 

Time (t). 

Number (N). 

All the texts and questions that will be provided in a questionnaire shall be checked carefully since the chosen wording 
might have a biasing influence on the outcome of the assessment. The questions in the questionnaire shall reflect the 
definition and the objective of the parameters described in EG 202 843 [i.l]. 

The questionnaires may be similar for the expert and customer, even if some complementary (more technical) questions 
may be added for experts. 



B.1 Opinion Rating (OR) 



Different types of scales may be used by customers or experts asked to give their opinion. As indicated in 

EG 202 843 [i.l] it is recommended to implement unipolar scales with 7 discrete ratings (preferably from to 6). 

All the possible selections shall be associated with textual description, and preferably for the terms associated to 
minimum rating (0) and maximum rating (6). Some examples of such terms are listed below. 

For the most of "OR" parameter the question could be defined as follows: 

"How do you rate <the question based on the definition of the parameter>?" 

When needed an additional information can be added as follows "Take into consideration . . .", or "e.g. . . .". 

Some examples may be found in informative annex D. 
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Table B.1 : Example of terms associated to minimum rating (0) and maximum rating (6) 



Term for minimum rating (0) 


Term for maximum rating (6) 


Applicable 
parameters 


No information 


All relevant information to make an informed 
judgement 


P101, P102 


Confusing, ambiguous and unclear 
representation of the terms of contract 


True and fair view of all relevant terms of 
contract 


P201 


Not comprehensible / Unclear and 
confusing 


Comprehensible and clear 


P615 


Inefficient and disorganised 


Professionally managed to meet needs of the 
customer 


P412, P512, P616, 

P652, P667, P672, 

P711, P815, P905, 

P913 


No scope/inflexible 


Flexible to meet customer's need 


P203, P204 


No assurance 


High level of assurance 


P666 


No empathy 


High level of empathy 


P666 


No responsiveness 


High level of responsiveness 


P666 


No sign of competence 


Competence at professional level 


P513 


Quite unprepared 


Fully prepared 


P513 


Never fulfils solution first time. Long 
drawn out affair to fulfil solution. 


Fulfilment of solution every time first time 


P624, P644 


Incorrectness in documentation 


Documentation correct and trustworthy 


P613 


Documentation incomplete 


Documentation has covered all relevant topics 
and is complete 


P613 


Never on time. Totally unreliable 


Always on time and can be relied 


P704 


Unorganised and chaotic 


Professionally organised and reliable 


P705 



B.2 Rate (%) 



The rate is expressed as a percentage calculated over the whole data. The individual answers are currently bipolar as 
YES or NO. 

The question asked shall be derived from the parameter definition as available in EG 202 843 [i.l]. 

The context of the question has to be defined and expressed as "Concerning your latest . . .". 



B.3 Time (t) 



The context (conditional) of the action is firstly defined; the customer or the expert is asked to define the time 
corresponding to the definition of the parameter. 

To avoid misunderstanding or mistakes the start and stop points of time shall be defined clearly. 

It is also needed to define the timeout condition for each time parameter. It should be taken into account that the error 
rate increases significantly if timeout values are chosen. 

When different time domains are defined in the questionnaire, it shall be needed to define clearly the way to 
aggregate the data across these different time domains (e.g. order of building averages per day, week, month, year). 

The questions may begin as: 

• "How many (Months/Day s/hours/Minutes) . . . <action from the SP>. . . " 
Alternative could be: 

• "How much time did pass between. . . ", "'What was the delay. . . " ; "what was the delay between. . . " 

For the reply several boxes may be provided, each defining a time slot. Such an approach simplifies the data input and 
analysis. However, it is also possible to leave the possibility for the participants to give the exact value (see clause B.6). 
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The number of "answering boxes" should be as Hmited as possible. 

B.4 Number (N) 

Each Customer or expert has to provide a number. So the question will begin as "How many. . ." 

EXAMPLE: "How many ways are currently used to transfer some kind of information?" 

It may be useful to ask supplementary open questions that are needed for a detailed report. An example of such 
questions is provided as: 

• "Which ways are used by service providers? 

• Which ways are usual for a private / business customer?" 

For the reply several "boxes" may be provided, each defining a number or a range of numbers. 

Such an approach simplifies the data analysis. 

However, it is also possible to leave the possibility for the participant to give the exact value (see informative annex D). 

The number of "answering boxes" should be as limited as possible. 

B.5 Other elements to be added in the questionnaires 
B.5.1 Pre-conditions 

Some parameters concern services that customers might not have used. Then customers should skip the question 
concerning the assessment of such parameters. Unfortunately, the experience shows that the customers usually rate the 
service they did not use. 

In order to avoid this, it is recommended for questionnaires dedicated to customer panels, to ask first if the service to be 
rated, was used at least once by the customer. This initial question forms a pre-condition to apply then the questions 
related to the parameters to be rated. 

For instance, parameters P661 to P667 concerning the complaint management, apply only if the customer has used the 
complaint management at least once. Thus the following question shall be asked during a customer survey: 

• Did you already use the complaint management desk? 

• Answering "yes" is the required pre-condition to proceed further with the P661 to P667 rating. 
In table D.l, a column indicates, if a pre-condition is required. It also proposes a text for the question. 



B.5.2 Open questions 



As indicated in the previous clauses, it may be useful to ask open questions to the customers or experts for further 
analysis. 

Such open questions or comment fields are useful: 

• Wherever the customers might have additional experience on specific topics. 

• Whenever a short description of some situation has more value than just a pure rating. 

It can be also useful to give the possibility to the participant to quote "no opinion". 

It may also be useful to leave the possibility to the participants to provide more details or comments (a special box to be 
provided), e.g. "Do you have comments on this?" 
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B.6 Validation of the questionnaire 

Before beginning the campaign the questionnaire shall be validated by an expert team (at least 3 experts) to ensure that 
the questions are unambiguous, fully comprehensible, relevant in the context and for the objectives of the campaign and 
that the questionnaire is well balanced. 



B.7 Questionnaires addressed to attendees 

When the questionnaires have been validated according to clause B.6, the questionnaires are made available to the 
attendees, through on line questionnaires, telephone calls, post mails or direct contact e.g. in commercial areas. 

It shall be checked that the questionnaires are replied by the attendees that have been identified by the selection process 
and not other people. 



B.8 Examples from the experiment made by STF 374 

To check the process described in the present document, STF 374 has organised a customer survey on a set of customer 
relationship stages and on the four types of parameters. The details of this experiment are available in annex D. 

The selected stages for these experiments are: 

• Preliminary information. 

• Contract establishment. 

• Complaint management. 

• Metering/Charging/Billing. 



Even if EG 202 843 [i.l] does not recommend to assess all the parameters of these stages through a Customer Survey, a 
questionnaire has been designed including all the parameters of the four stages. The selected set of parameters is highly 
representative of the questions to be asks for all the parameters defined in EG 202 843 [i.l]. They can be used as 
references when developing future assessment campaigns. 
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Annex C (informative): 

Questions related to the user's background 

It is very important to ensure that the information collected in this aspect during the survey is kept confidential. 

C.1 Questions related to the user's background 
C.1.1 Personal data 

Gender: D Female D Male 

Age: <20 years D 21-30 years D 31-40 years D 41-50 years D 51-60 years D 61-70 years n>70 years D 

Language (s) (native) learnt 

Profession/Education: 

Current residence: 

Incomes/Year (euros): <10k D 10k - 25k D 25k - 50k D 50k - 100k D >100k D 
How many years of using telecommunications services do you have? 

<1 year D 1-3 years D 3-10 years D >10 years D 

Do you have any disability: visual D auditory D tactile D 

Phone number: Fixed Mobile 

Email address 

C.1 .2 How often do you use ICT services on an average (several 
possible answers)? 

Phone calls D 

almost all the time D 1 to 3 times a day D almost every day D 1 to 5 times a week D less than 5 times a month D 
SMSD 

almost all the time D 1 to 3 times a day D almost every day D 1 to 5 times a week D less than 5 times a month D 
IPTV or Mobile TVD 

almost all the time D 1 to 3 times a day D almost every day D 1 to 5 times a week D less than 5 times a month D 
VoD D 

almost all the time D 1 to 3 times a day D almost every day D 1 to 5 times a week D less than 5 times a month D 
AudioA^ideo Streaming D 

almost all the time D 1 to 3 times a day D almost every day D 1 to 5 times a week D less than 5 times a month D 
Email communication D 

almost all the time D 1 to 3 times a day D almost every day D 1 to 5 times a week D less than 5 times a month D 
Audio/video calls/conferences D 

almost all the time D 1 to 3 times a day D almost every day D 1 to 5 times a week D less than 5 times a month D 
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Web browsing D 

almost all the time D 1 to 3 times a day D almost every day D 1 to 5 times a week D less than 5 times a month D 



C.2 What kind of ICT are you contracted with your 
Service Providers - (several possible answers)? 



Fixed Hne (PSTN or ISDN) D less than 1 month D 



Fixed line (VoIP) 
Mobile phone 
Internet access 
IPTV/Mobile TVA^oD 
Other: 



D less than 1 month D 

D less than 1 month D 

D less than 1 month D 

D less than 1 month D 

D less than 1 month D 



1 year D 
1 year D 
1 year D 
1 year D 
1 year D 
1 year D 



more than 3 years D 
more than 3 years D 
more than 3 years D 
more than 3 years D 
more than 3 years D 
more than 3 years D 



C.3 What are the terminals you are currently using 
(several possible answers)? 



Fixed phone (PSTN, VoIP or ISDN)-including DECT D 

Mobile phone D 

PDA D 

TV set D 

Personal computer D 

Public internet terminal D 

Game console D 

Other: D 



C.4 What kind of access (several possible answers) for 
internet, if you know them? 



Ethernet 


D 


ADSL 


D 


Mobile 


D 


DSL 


D 


Wimax 


D 


PSTN 


D 


Cable 


D 


FTTH 


D 


Other: 


D 
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C.5 Service providers 

C.5.1 With how many service providers do you have contracts for 
your services? 

1 D Name 

2 D Names 

3 D Names 

More than 3: D Names 



C.5. 2 Have you moved at least once from a service Provider to 
another one? 



For what reason(s)? . 



C.5.3 Do you use ICT for 

Private usage D Business D Both D 



C.6 Do you have any additional information to provide? 
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Annex D (informative): 

Example of a campaign performed by STF 374 

Context of the questionnaire defined by the STF 374: 

• The experiment did not address any specific service but was intended to receive a global feedback from 
members of a customer association on the perceived QoS from their service providers for several customer 
relationship stages. 

• The questions developed for this specific experiment may only be considered as examples that can be adapted 
when developing a questionnaire in another environment and with another purpose: 

The following questions have been asked to the participants to the experiment conducted by the STF 374. 
They may be used as example to create new questionnaires in particular for parameters that are not 
defined in the following tables. 
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Table D.I: Opinion Rating 



Parameter 


Type 


Title 


Precondition: Yes/No 


Question 


P101 Parti 


OR 


Preliminary Information (PI) 
Integrity of preliminary 
information: Content 


No 


Content: Were all the relevant information provided as you expected? 


P101 Part 2 


OR 


Preliminary Information (PI) 
Integrity of preliminary 
information: Language 


No 


Language: Was the information provided clear and understandable without 
any ambiguity? 


P101 Parts 


OR 


Preliminary Information (PI) 
Integrity of preliminary 
information: Style 


No 


Style: How would you rate the overall style, presentation and professionalism 
of the preliminary information provided? (Take into consideration issues like 
typeface, background colours, contrast, acoustics, display choreography etc.) 


P102 


OR 


Pricing transparency 


No 


Did you find the pricing information comprehensible? 
(see note) 


P201 


OR 


Integrity of contract information 


No 


How would you rate the integrity of the contractual document? (Take into 
consideration issues like maintenance, performance levels, invoicing, 
warranty conditions, compensation schemes, cancellation procedures, 
cessation, etc.) 


P203 


OR 


Flexibility for contract 
customization before contract 


No 


How would you rate the flexibility of your service provider to customise the 
contract before signature e.g. by applying options? 


P204 


OR 


Ease and flexibility to amend 
terms after formal contract 


No 


How would you rate the flexibility of your service provider to further adapt the 
contract after signature e.g. by applying options? 


P664 


OR 


Integrity of complaint resolution 


Yes: Did you already use the complaint 
management desk? 


Concerning your latest accepted complaint; Was your complaint resolved 
correctly? 


P665 Part 1 


OR 


Customer perception of 
complaint management: 
Assurance 


Yes: Did you already use the complaint 
management desk? 


How would you rate the service provider's complaint management related to 
assurance at all? (Take into consideration characteristics like competence, 
engagement, credibility and trust.) 


P665 Part 2 


OR 


Customer perception of 
complaint management: 
Empathy 


Yes: Did you already use the complaint 
management desk? 


How would you rate the service provider's complaint management related to 
empathy at all? (Take into consideration characteristics like ease of contact, 
market awareness, listening to customers and keeping the customer 
relation.) 


P665 Part 3 


OR 


Customer perception of 
complaint management: 
Responsiveness 


Yes: Did you already use the complaint 
management desk? 


How would you rate the service provider's complaint management related to 
responsiveness at all? (Take into consideration characteristics like 
willingness, promptness of reaction, estimation of realistic timeframes and 
follow-up contacts where necessary.) 


P666 


OR 


Overall quality of the complaint 
management process 


Yes: Did you already use the complaint 
management desk? 


How would you rate the overall handling of the complaint management 
process? (Take into consideration the accessibility to the service, the correct 
resolution at the first attempt, the speed of resolution and the organisational 
competence of your service provider.) 


NOTE: For further assessment campaign, it should be changed as "How would you rate the pricing information comprehension?" 
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Table D.2: Percentage 



Parameter 


Type 


Title 


Precondition: Yes/No 


Question 


P103 


% 


Availability of Preliminary 
Information (PI) 


No 


Could you retrieve the preliminary information easily? (Take into 
consideration different information sources like web, email, flyers, phone calls 
and combinations of them.) 


P202 


% 


Compliance of contractual terms 
with preliminary information (PI) 


No 


Was the contract document compliant to the previously provided preliminary 
information? 


P661 


% 


Accessibility of the complaint 
management desk 


Yes: Did you already use the complaint 
management desk? 


Concerning your latest attempt to access the complaint management desk of 
your service provider: Did you succeed in accessing it? 


P662 


% 


Recognition of the customer 
complaints 


Yes: Did you already use the complaint 
management desk? 


Concerning your latest complaint to the complaint management desk of your 
service provider; Was your complaint accepted? 


P663 


% 


Complaint solutions not complete 
and correct first time 


Yes: Did you already use the complaint 
management desk? 


Concerning your latest accepted complaint; Was the complaint solved to your 
satisfaction at the first attempt by the service provider? 


P664 


% 


Integrity of Complaint resolution 


Yes: Did you already use the complaint 
management desk? 


Concerning your latest accepted complaint; Was the complaint finally solved 
to your satisfaction by the service provider? 


P801 


% 


Accessibility of the tariff 
information 




Concerning your latest attempt to access your provider's tariff information; 
Were you able to access the tariff information? 


P802 


% 


Successful notification of 
exceeding billing budget 


Yes: Did you subscribe the notification 
of exceeding the billing budget? 


Concerning your latest exceeding of budget; 

Were you notified accordingly when you exceeded your budget? 


P804 


% 


Accessibility of the account 
management 


No 


Concerning your latest attempt to access the account status at your service 
provider: Did you succeed in accessing it? 


P806 


% 


Timeless of bill delivery 


No 


Were all the bills throughout the last 6 months, available at the expected 
time? (availability is as applicable per post mail, email or uploaded on your 
web account). Did you receive all the expected bills throughout the last 6 
months? (see note) 


P808 


% 


Late notification of amount due 


No 


Considering your latest direct debit payment; Has the bill been received 
before the direct debit was executed? 


NOTE: Please indicate the expected time (e.g.: number of days before direct debit occurs, number of days before the payment). 
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Table D.3: Time and number 



Parameter 


Type 


Title 


Precondition: Yes/No 


Question 


P104 


t 


Response time for the provision 
of Preliminary Information (PI) 


No 


FOR EXPERTS ONLY: Please answer only if you retrieved the preliminary 
information via web access; How much time did pass between the request 
and the delivery of preliminary information? (Please express in minutes, hours 
or days.) 


P803 


t 


Notification time (delay) of 
exceeding billing budget 


No 


FOR EXPERTS ONLY: If you are using a notification service when you reach 
a predefined budget level; What was the period of time between exceeding 
the billing budget and the reception of the according notification? 


P805 


t 


Time to update charging 
information 


No 


FOR EXPERTS ONLY: What was the delay of the charging information on 
your account? 


P807 


t 


Bill delivery delay 


Yes: Did you experienced delays in the 
availability of bills? 


How many days was the availability of the bill delayed? (availability is as 
applicable per post mail, email or uploaded on your web account) 


P809 


N 


Modes of billing information 
transfer 


No 


How many ways do you have to access your accounting information? 


Table D.4: Open additional question 


Parameter 


Type 


Title 


Question 


PGeneral 




General question 


Do you consider the four parameters above deal with most, if not all of the pertinent features 

of PI from the customer perspective? 

If not what additional parameter would you like to see included? 
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Annex E (informative): 

Examples of tasks for Expert panels 

The table E.l defines the parameters that are intended to be checked by the Expert Panel. 

NOTE: For information, the different means defined for assessment campaigns are kept in the table: "CS" stands 
for "Customer Survey", "EP" stands for "Expert Panel" and "SPDA" stands for "SP Data Analysis". 

In the table E.l the fifth column (Assessment process.) lists all the assessment methods but the last column applies only 
to expert and trained customer panels. 

For the experts, two approaches are described in the table E.l: 

Tl When possible the experts act as an anonymous customer (e.g. request Preliminary information, contact the 
service desk,. . .), but without needing any contract with the SPs. 

For parameters based on assessment of SP services or of SP data, the experts have to plan appointments with 
SP representatives or managers and are clearly identified as the experts in charge of QoS testing. In such cases 
or in complement they should have access to some data stored by the SP and have to sign confidentiality 
agreements. 

T2 The experts subscribe to different contracts (with the SP involved in the assessment campaign and for the 
services defined in the scope of the campaign). In this case, financial provision should be planned for the 
expenses due to these contracts. The experts will act as real customers, for the different stages of the 
relationship: in such a case, the campaign should be long enough to provide significant results for the 
parameters defined for the campaign (at least three months). The experts could report periodically (e.g. every 
month). 

Whatever the approach implemented, each expert reports individually the results or opinions for each parameter, for 
each medium used and each SP under test. 

All the experts involved in an experiment are invited for a common meeting, during which each expert reports on the 
results and comments and discuss them in order to produce a global value for the parameters defined in the campaign. 
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Table E.1 : Parameters intended to be checked by the Expert Panel 









3 fi) 


> 




Stage 


Parameter name 


So 




(D '£. 

ip 3 


Proposed Task for Expert or 
trained customer Panels 






(D -^ 


-« (D 


W <D 








^ 


^ 


s 




Preliminary 




















For T1 and T2 approaches, the 


information 










experts should consider the PI 
provided (or available) by each 
service provider through one 
selected mode (e.g. Web page, 
flyer, contact with the service desk). 
This task could be repeated for 
each of the available mode. 
The expert provides a rating for 




Integrity of preliminary 
information (PI) 


OR 


101 


EP 


each of the three parameters 
qualifying the integrity of the 
preliminary information as 
described in EG 202 843 [i.1]. 
Each expert reports individually for 
each of them (for each mode and 
each SP). 

An "aggregated" OR may be 
provided by an experts' meeting as 
an additional result (associated to 
comments from the experts). 










For T1 and T2 approaches, the 












experts should analyze all the tariffs 












for the services currently offered (or 












for a selected part of them, 












e.g. mobile including flat rate, 












roaming) and have to give an 




Pricing transparency 


OR 


102 


CS-EP 


opinion on their perception of 
pricing transparency. 

During an expert meeting the 
results and comments provided by 
the experts are discussed. 
An "aggregated" OR may be a 
result of the experts' meeting. 


Response time for the 


t 


104 


CS-EP 


Experts have to record the time to 




provision of PI 




provide PI for each mode. 


Contract 




















For T1 approach, the experts 


establishment 










should consider the contract 
documentation provided by each 
Service provider through one 
medium (e.g. Web page, flyer, 
contact with the service desk). 




Integrity of the contract 


OR 


201 


TrainpH H-FP 


To achieve that the experts have to 




information 








receive from the different SPs the 
draft contract for the services under 
assessment. 

An "aggregated" OR may be a 
result of an expert meeting 
(see note 1). 
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Stage 



Parameter name 



fi) 



3 (b 



3 fi) 

ii 



> 

0) 
(D 
0) 
0) 

3 



3 



Proposed Task for Expert or 
trained customer Panels 



Compliance of 
contractual terms with PI 



% 



202 



Trained C-EP 



The expert may look specifically for 
compliance in the PI with the 
information provided in the contract. 
The expert panel will have an 
insight into the legal aspects of the 
use of this service or family of 
services to enable them to critically 
evaluate the legal aspects and from 
the customer's and SP's point of 
view. 



Flexibility for 
customization before 
contract 



OR 



203 



Trained C-EP 



For T1 approach, the experts 
should contact the service desk of 
the different SPs under assessment 
and collect answers to a set of 
predefined questions (e.g. about 
additional service features, service 
performance, specific 
conditions/prices). 



Service 
provisioning 



Contract cancelled due 
to non fulfilment 



% 



304 



EP 



Completeness of 
fulfilment of contractual 
specification in the 
provision of a service 



For T1 approach, the experts 
should collect the data from the SPs 
under assessment. As it may be a 
sensitive issue, the expert should 
not use "mystery call" but be 
indentified and should contact the 
relevant SP Managers or Teams in 
charge of these issues. 



305 



CS-EP 



Punctuality of service 
provisioning 



306 



CS-EP 



"CS" should be the preferred 
method. However if an expert panel 
is used, the best approach for this 
parameter should be T2. The 
experts should, sign all the needed 
contracts and should check the 
provisioning as defined in the 
contract, (as indicated above, this 
approach may be expensive 
because the experts have to 
subscribe to offers. The type of 
contract to be signed should be 
defined in advance (e.g. for the 
most popular services offered by 
the different SPs). 

Alternative task: 

The experts should contact the SP 

service desks, to ask for relevant 

managers and to ask for replies to 

dedicated questions about service 

provisioning. 



Provisioning not 
complete and correct first 
time 



% 



308 



CS-EP 



For T2 approach, the experts, after 

having subscribed to offers should 

report on the actual provisioning for 

each contract. 

For T1 approach, the experts 

should collect data from the SP and 

report on this collection 

(see note 2). 
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Stage 


Parameter name 


Type of 
parameter 


3? 

ii 


il 

i| 

0) 3 

3 


Proposed Tasl< for Expert or 
trained customer Panels 


Service alteration 












Technical reliability of 
service within an agreed 
period after alteration 


% 


408 


SPDA-CS-EP 


For T1 approach, the experts 
should contact the service desk 
management and ask about the 
number of cases treated during a 
dedicated period and about the 
possible failures. 

Each expert reports on the results 
collected. 


Response time of the 
alteration service 


T&% 


409 


SPDA-CS-EP 


For T1 approach, this task should 
be coupled with P408. 
During the interview described for 
P408 the experts should ask 
questions about the response time 
stored by SP. 


Organisational efficiency 
of service provider to 
carry out service 
alteration 


OR 


412 


SPDA-EP 


Experts should obtain relevant data, 
where available, from the SP and 
make an informed judgement in 
other cases to arrive at an OR 
value. 


Teclinical upgrade 












Organisational efficiency 
of service provider to 
carry out technical 
upgrade 


OR 


512 


SPDA-CS-EP 


For T2 approach, the experts 
should report on actual experiments 
they conducted (relevant scenarios 
should be defined, based on recent 
upgrades offered by the SPs). 
If they have subscribed to SP offers 
the upgrades will be based on the 
contracts effectively subscribed. 

For T1 approach, the experts 
should also contact the SP 
management to ask for a detailed 
presentation of the process 
implemented to carry out technical 
upgrade. Questions about the 
number of cases to be treated, the 
time needed to carry out technical 
upgrades may be also asked. 

Each expert provides the effective 
(forT2 approach) or collected (for 
T1 approach) results. 


Competence and 
preparedness of service 
provider for technical 
upgrade 


OR 


513 


SPDA-EP 


This parameter will be assessed in 
a similar way as P51 2 and 
preferably at the same time and 
with the same process. 
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Stage 



Parameter name 



fi) 



3 (b 



3 fi) 

ii 



> 

0) 
(D 
0) 
0) 

3 



3 



Proposed Task for Expert or 
trained customer Panels 



Integrity (correctness and 
completeness) of 
documentation 



OR 



613 



CS-EP 



Service Support ■ 
Documentation 



For T1 and T2 approaches, the 
experts should consider the 
documentation provided by each 
service provider through one 
medium (e.g. Web page, flyer, 
contact with the service desk). This 
task could be repeated for all the 
other modes. The expert will 
consider the different parameters 
described in EG 202 843 [i.1] for 
integrity of documentation and will 
report on each of them, for each 
mode and each SP. 

During an expert meeting the data 
are discussed and an aggregated 
OR may be a result (associated to 
comments from the experts). 

An aggregated OR may be an 
additional result obtained after 
discussion during experts meeting. 



Modes of documentation 



614 



SPDA-EP 



For T1 and T2 approaches the 
experts should collect all the 
available documentations and 
report the number. 



Legibility of 
documentation 



OR 



615 



EP 



For T1 and T2 approaches, the 
experts should give their opinion on 
the legibility of each documentation 
they have collected in P614. 

An aggregated OR may be a result 
obtained after discussion during 
experts meeting. 



Overall reliability of 
documentation services 



OR 



616 



EP-CS 



For T1 and T2 approaches, the 
experts should give their opinion on 
the overall reliability of all the 
documentation collected in P614 
(SPs may also be asked for 
complementary information when 
needed). 

An aggregated OR may be an 
additional result obtained after 
discussion during experts meeting. 



Service Support - 
Technical support 



For an expert panel the best 
approach should be to have an 
appointment with the technical 
support of each SP. The expert will 
also provide personal views based 
on the answers received from the 
SP and possibly by analysing data 
available by SP. 

For T2, personal observations 
linked with the contracts subscribed 
could also be taken into account 
(see note 3). 
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1^ 


35" 


^r 




Stage 


Parameter name 


3 (D 

So 


^1 


n 


Proposed Task for Expert or 
trained customer Panels 






(D -K 


-« (D 


(/) (D 








-^ 


-^ 


3 














For such a parameter a Customer 












Survey should be preferred. (The 












Expert Panel may not be the better 












approach). 




Accessibility to the 
technical support 


% 


621 


SPDA-CS-EP 


The experts are invited to contact 
the technical support of all the SPs 
under assessment and to report 
about the accessibility to it. The 
experts should take benefit of these 
contacts to ask all relevant 
questions. 










During the interview of the SP (or 




Technical solutions 








analysis of Data stored by SP), as 




achieved within a 


% 


622 


SPDA-CS-EP 


described for P621 the expert 




specified period 








should collect relevant data on this 
parameter. 










Based on answers received from 












SP to questions raised during the 












interview(s) described for P621 and 












after crosschecking with the data 












made available by SP, the Experts 




Integrity of technical 
solutions 


OR 


624 


CS-EP-SPDA 


should provide a rating. 










For T2 approach, the experts could 












also take into account the actual 












situations met within the contracts 












subscribed with the different SPs 












The experts will meet to provide a 












consolidated rate. 










After interview as defined for P621 












the experts should provide a result 




Reliability of technical 
solutions achieved 


% 


625 


SPDA-CS-EP 


based on data provided by the SP. 
For T2 approach, if the experts 
need to request technical solutions 
for the subscribed contracts, they 
may also provide actual comments. 










For T1 and T2 approaches the 












experts should look for the different 












modes and report. 












During the interview of the SP (or 




Modes of technical 


N 


fipfi 


QpnA_FP 


analysis of Data stored by SP), as 




support 


i\ 


vJ^vJ 


or L^AA ^r 


described for P641 the experts, 
should collect relevant data on this 
parameter Each expert is intended 
to look for the different modes and 
provides the results of this analysis. 
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Stage 



Parameter name 



fi) 



3 (b 



3 fi) 

ii 



> 

0) 
(D 
0) 
0) 

3 



3 



The process described for 
Technical support applies for 
commercial support. 

For an expert panel, the best 
approach should be to have an 
appointment (interview) with the 
commercial support of each SP. 
The expert will also provide his 
personal view based on the 
answers received from the SP and 
possibly by analysing data available 
bySP. 

For T2, personal observations 
linked with the contracts subscribed 
could also be taken into account. 

Each expert provides the results 
and data are discussed within an 
experts meeting (see note 4). 



Proposed Task for Expert or 
trained customer Panels 



Service Support ■ 

Commercial 

support 



Accessibility of the 
commercial support 



% 



641 



SPDA-CS-EP 



For such a parameter, a Customer 
Survey should be preferred (the 
Expert Panel may not be the better 
approach). 

For T1 approach the experts should 
contact the commercial support of 
all the SPs under assessment and 
to report about the accessibility to it. 



Commercial solutions 
achieved within a 
specified period 



643 



SPDA-CS-EP 



During the interview of the SP (or 
analysis of Data stored by SP), as 
described for P641 the expert 
should collect relevant data on this 
parameter. 



Integrity of solution 
achieved by SP 



OR 



644 



CS-EP-SPDA 



Based on answers received from 
SP to questions raised during the 
interview(s) described for P641 , and 
after cross checking with the data 
made available by SP, the Experts 
should provide a rating. 

For T2 approach, the experts could 
also take into account the actual 
situations met within the contracts 
subscribed with the different SPs 
The experts will meet to provide a 
consolidated rate. 



Modes of commercial 
support 



645 



SPDA-EP 



For T1 and T2 approaches, the 
experts should look for the different 
modes of technical supports and 
report. During the interview of the 
SP (or analysis of Data stored by 
SP), as described for P641 the 
experts should collect relevant data 
on this parameter. Each expert is 
intended to look for the different 
modes and provides the results of 
this analysis. 
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ii 
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0) 
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3 



Proposed Task for Expert or 
trained customer Panels 



Organisational efficiency 
of commercial support 



OR 



652 



SPDA-CS-EP 



After having conducted the 
interviews defined above and 
having analysed the data provided 
by the SPs, experts may provide a 
personal opinion rating about the 
organisational efficiency. 
The rating is explained by objective 
results or observations done by the 
experts. 



Service Support ■ 

Complaint 

management 



For an expert panel the best 
approach should be to have an 
appointment (interview) with the 
complaint management department 
of each SP under assessment. The 
experts should also provide 
personal views based on the 
answers received from the SP and 
possibly by analysing data made 
available by the SP (see note 5). 
For T2 approach, if the experts face 
such problems they may report on 
these actual observations. 



Accessibility of the 
complaint management 
desk 



661 



SPDA-CS-EP 



The Expert Panel may not be the 
better approach and a Customer 
Survey should be preferred. 

However, for experts involved in the 
assessment process, the task for 
each expert to contact the 
complaint management (and if 
possible a manager of this 
department) of the SP and to report 
about the accessibility (in particular 
the waiting time, the number of 
calls,..). 

The experts should take benefit of 
accessing the complaint 
management desk to ask a set of 
questions (interviews) as defined for 
other parameters. 



Recognition of the 
customer complaints 



662 



SPDA-CS-EP 



For T1 and T2 approaches, the 
experts should analyze the data 
made available by the SPs under 
assessment and the information 
collected during interviews. 

For T2 approach, the experts 
having contacted the complaint 
management desk could also report 
on these actual observations. 
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Proposed Task for Expert or 
trained customer Panels 






n ■* 


-« (D 


(/) (D 








-I 


-^ 


3 














Based on answers received from 












SP to questions raised during the 












interview(s) conducted by the 












expert, as defined for P661 and 












after crosschecking with the data 












made available by SP, the experts 












should provide a ratio. 












The experts have to provide a 




Integrity of complaint 


% 


664 


SPDA-CS-FP 


consolidated rating (by explaining 




resolution 






\ji L^r\ \j\j 1 1 


the ratio with objective 
observations). 

For T2 approach, if the experts 
contacted the complaint 
management desk for issues linked 
with some subscribed contracts 
they could also report on these 
actual observations. 










For this parameter, the preferred 












method is a Customer Survey. 












However, the experts, as very 




Customer perception of 
the complaint 
management 


OR 


665 


CS-EP 


experienced customers (and in 
particular those involved in T2 
approach) they may provide a 










rating, based on the results and 












observations done during interviews 












or actual cases, as defined for 












P661. 










Experts provide a rating based on: 












- the answers provided by SP 












during interviews, as defined for 




Overall quality of the 








P661 




complaint management 


OR 


666 


CS-EP 


- Analysis of data provided by SP 




process 








- and when available their personal 
opinion when contracts have 
been subscribed for the 
assessment campaign. 










After having conducted the 












interviews defined for P661 and 












having analysed the data provided 




Organisational efficiency 








by the SPs, experts may provide a 




of complaint 


OR 


671 


SPDA-CS-EP 


personal opinion rating about the 




management system 








organisational efficiency. 

The rating is explained by objective 

results or observations done by the 

experts. 
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3 



Proposed Task for Expert or 
trained customer Panels 



Metering, 
Charging, Billing 



The best approach is a "Customer 
survey". 

For T2 approach, each expert 
having subscribed contracts with 
different SPs is invited to report on 
the actual situations observed 
during several months. With such 
an approach the experts are 
intended to act in a similar way as 
any customer (but using their high 
level knowledge to ask pertinent 
questions). 

Due to the low number of results 
from each Expert, results and data 
are discussed within an experts 
meeting. 

For T1 approach, experts should 
organize an appointment with the 
SPs and collect relevant data. 



Accessibility to the tariff 
information 



% 



801 



SPDA-CS-EP 



For T1 and T2 approaches, experts 
should collect tariff information 
available by different means (web, 
flyer, phone call, etc.) for each SP 
and for the most popular services 
currently offered. 



Successful notification of 
exceeding billing budget 



% 



802 



SPDA-CS-EP 



For T2 approach, experts should 
report about the actual observations 
concerning successful notifications. 

For T1 approach, experts should 
report about the data provided by 
SPs on this parameter. 



Notification time (delay) 
of exceeding billing 
budget 



803 



SPDA-CS-EP 



This parameter should be better 
assessed by a customer survey. 

However, if an expert panel is 
involved in the assessment, the 
tasks to be done are similar to 
those defined for P802: 

For T2 approach, experts should 
report about the actual observations 
concerning successful notifications. 

For T1 approach, experts should 
report about the data provided by 
SPs on this parameter. 
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Proposed Task for Expert or 
trained customer Panels 



Accessibility to the 
account management 



% 



804 



SPDA-CS-EP 



This parameter should be better 
assessed by a customer survey. 

However, if an expert panel is 
involved in the assessment, the 
tasks to be done are similar to 
those defined for P802: 

For T2 approach, experts should 
report about the actual observations 
concerning successful notifications. 

For T1 approach, experts should 
report about the data provided by 
SPs on this parameter. 



Time to update charging 
information 



805 



SPDA-CS-EP 



This parameter should be better 
assessed by a customer survey. 

However, if an expert panel is 
involved in the assessment, the 
tasks to be done are similar to 
those defined for P802: 
For T2 approach, experts should 
report about the actual observations 
concerning successful notifications. 

For T1 approach, experts should 
report about the data provided by 
SPs on this parameter. 



Late notification of 
amount due 



808 



SPDA-CS-EP 



This parameter should be better 
assessed by a customer survey. 

However, if an expert panel is 
involved in the assessment, the 
tasks to be done are similar to 
those defined for P802: 
For T2 approach, experts should 
report about the actual observations 
concerning successful notifications. 

For T1 approach, experts should 
report about the data provided by 
SPs on this parameter. 



Modes of billing 
information transfer 



809 



SPDA-EP 



The tasks to be done are similar to 
those defined for P802: 
For T2 approach experts should 
report about the actual observations 
concerning successful notifications. 

For T1 approach experts should 
report about the data provided by 
SPs on this parameter. 



Organisational efficiency 
of the billing service 



OR 



815 



SPDA 
CS-EP 



After having conducted the 
interviews defined above and 
having analysed the data provided 
by the SPs, experts may provide a 
personal opinion rating about the 
organisational efficiency. 
The rating is explained by objective 
results or observations done by the 
experts. 
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Parameter name 


Type of 
parameter 
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ii 


ll 
i| 

0) 3 
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Proposed Task for Expert or 
trained customer Panels 


Network/Service 

Management by 

the customer 










A specific task could be given to 
each expert of the expert Panel, for 
each SP. Due to the low number of 
results from each Expert, results 
and data are discussed within an 
experts meeting. 


Frequency of outages 


% 


902 


SPDA-EP 


For T1 approach experts should 
report about the data provided by 
SPs on this parameter. 


Overall reliability of 
network/service 
management service 


OR 


905 


CS-EP 


This parameter should be better 
assessed by a customer survey. 

However for T2 approach experts 
should report about the actual 
observations concerning the overall 
reliability. 


Organisational efficiency 
of network/service 
management service 


OR 


913 


CS-EP 


After having conducted the 
interviews defined above and 
having analysed the data provided 
by the SPs, Experts may provide a 
personal opinion rating about the 
organisational efficiency. 
The rating is explained by objective 
results or observations done by the 
experts. 


Cessation 










To question SPs on cessation, 
results and data are discussed 
within an experts meeting. 


Cessation request 
acknowledgement 


% 


1002 


CS-SPDA-EP 


For T2 approach, the experts 
should act as real customers, on 
the basis of the contractual terms of 
the contract they subscribed. 

For a T1 approach, experts should 
analyze the data stored by SP. 


Accessibility of the 
cessation facility 


% 


1003 


CS-SPDA-EP 


For T2 approach, the experts 
should act as real customers, on 
the basis of the contractual terms of 
the contract they subscribed. 

For a T1 approach, experts should 
analyze the data stored by SP. 


Contractual cessation 
achieved 


% 


1004 


CS-SPDA-EP 


For T2 approach, the experts 
should act as real customers, on 
the basis of the contractual terms of 
the contract they subscribed. This 
task may only be done after the 
time period dedicated to the whole 
experiment. 

For a T1 approach experts should 
analyze the data stored by SP. 


NOTE 1 : The contract has to be asked. For some experiments the contract will not be signed and only analysed by 
the experts. For T2 approach the experts should sign the contracts and check carefully all the relationships 
with the SPs. 

NOTE 2: "CS" is a better method than "EP". 

NOTE 3: It could be useful to define some scenarios (e.g. by defining a set of services or configurations). 

NOTE 4: It could be useful to define some scenarios (e.g. by defining a set of services or configurations). 

NOTE 5: It could be useful to define some scenarios (e.g. by defining a set of services or configurations). 
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